Australian experiences using a COVID-19 chest radiography reporting template: An implementation study

Introduction: Chest X-rays are an important tool in COVID-19 disease management and progression.1 Several radiology societies have developed structured reporting templates to reduce interpretation variability and measure concordance.2 This study aimed to measure concordance of three international chest X-ray reporting templates in a Sydney hospital. Method(s): 12 radiologists viewed a test set of 50 COVID-19-positive patients' chest X-rays (30 classic appearance, 20 indeterminate) and 20 normal or 'other' diagnoses chest X-rays. Radiologists classified the cases according to the Royal Australian and New Zealand College of Radiology (RANZCR), British Society of Thoracic Imaging (BSTI) and modified Co-RADS (Dutch)3 templates. Intra-reader and inter-reader reliability were calculated plus measures of experiences of using templates. Result(s): Inter-reader agreement between radiologists was highest for the BSTI template (0.46), followed by RANZCR (0.36) and modified Co-RADS (0.31). The intra-reader agreement across the three templates for 'classic/characteristic' COVID-19 cases was 0.61, for 'normal' cases 0.76 and 'alternative' 0.68 with large variations that were not related to experience. Radiologists agreed the templates were easy to use and would consider using them in the future, although some cases had very low concordance (intra- and inter-reader). Conclusion(s): The BSTI template yielded highest agreement for reporting all chest X-ray types. There was a large range of intra-reader agreement for all four types of patient presentations. Further investigation of radiology lexicon is required to seek reasons for variation as well as understanding the perception of utility by referring physicians. Extension of this work should include radiographers using the templates.

The utility of COVID-19 reporting templates for chest radiography (CXR): an Australian single centre study

Purpose: Chest radiographs (CXRs) are an important tool for COVID-19 disease management and progression.1 A number of international thoracic/radiology societies have developed structured reporting templates to reduce interpretation variability and measure concordance.2 Recent research into COVID-19 templates have used small data sets, small numbers of radiologists and focused on predictive disease progression. The aim of this study is to evaluate the utility of three (Australian, British, Dutch) CXR qualitative reporting templates in a large Sydney teaching hospital. Methods and Materials: Twelve staff specialist radiologists were recruited to participate in the study. Retrospective CXRs between 2020-21 were collected and organized into test sets, containing the CXRs of 50 PCR-positive COVID-19 patients and classified as 'classic/ characteristic' for radiology signs of COVID-19 disease (30 cases) or 'indeterminate' for radiological signs (20 cases) by two expert radiologists. A further 10 CXRs of patients who were imaged for an 'alternative' diagnosis and 10 CXRs that had been reported 'normal' were included. Radiologists were assigned to one of three image sets, with the 70 images randomized and instructed to report the cases using three templates: the Royal Australian and New Zealand College of Radiology (RANZCR), British Society of Thoracic Imaging (BSTI) and modified Co-RADS (Dutch)3. Fleiss' Kappa Coefficient was used for agreement between radiologists on each template (inter-reader), and for radiologist's agreement within themselves using the three templates (intra-reader). Result(s): Inter-reader agreement between radiologists was highest for the BSTI template at 0.46 (moderate agreement), followed by RANZCR (0.36) and modified CO-RADS (0.31) (fair agreement). For all templates, agreement was highest for 'normal CXRs" and lowest for 'indeterminate', with moderate or fair agreement for the 'classic' COVID-19 cases (BSTI 0.44;RANZCR 0.31;mod-CO-RADS 0.31). The intra-reader agreement across the 3 templates for 'classic/characteristic' COVID cases was 0.61, for 'normal' cases 0.76 and 'alternative' 0.68 (all at substantial agreement), indicating that radiologists largely were consistent with classification of cases. Conclusion(s): The BSTI template yielded the highest agreement among radiologists for reporting COVID-19 CXRs as well as for 'normal' and 'alternative'. There was a large range of intra-reader agreement for all 4 types of patient presentations, however the level of agreement equal to or higher than 'moderate'. Structured reporting templates have yielded promising results for concordance between radiologists and reliability within radiologist's reporting of CXRs. Further investigation of radiology lexicon within templates is required to seek reasons for variation in concordance, as well as the demographics of readers.